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ABSTRACT 

This study examined current approaches to computer 
courseware evaluation and evaluated published models on courseware 
evaluation from the perspectives of end-user/project proposers, the 
commercial vendor , and the academic or non-aligned consultant. First, 
a list of primary sources of computer courseware evaluation were 
located and analy2ed. From this analysis, a list of criteria or 
criterion elements was derived; grouped into the global categories of 
use, quality, and efficiency; and cross-referenced with the primary 
sources. A checklist of the resulting criterion elements was 
formulated and mailed to a sample of 14 end-users (contract 
agencies), vendors (courseware developers), and academic-based 
educators or faculty, who were asked to rate the criteria on their 
importance in software evaluation. The respondents — five vendors, 
four users, and three academics — generally rated the infbrmation 
provided on both hardware and software, including documentation, as 
very important for doing an adequate courseware evaluation. No global 
category dominated, and no major differences were noted among 
vendors, users, and academics. In general, it was found that users 
were less rigorous than vendors or academics. Eleven general articles 
are listed, as well as the sources of courseware evaluation models 
used in designing this project, and charts showing participant 
responses. ( LMM) 
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Intr oduction * 

The genesis of this presentation can be linked directly to a paper 
delivered by Dr. Suzanne Sax of AT&T Business Service Education group at 
the 1983 NSPI Conference. Dr. Sax's presentation on a computer courseware 
Vendor Selection Model drew considerable audience interest and reaction — 
not all of it positive. As I sat in the audience, T observed at least 
three reactions. First, end-users Cprolect proposers) saw her mod^l as a 
wav to demystify the process of getting oualitv computer courseware for 
their applications. Second, commercial vendors (courseware developers) 
wanted to know how thev could g*»t on AT&T's list of potential bidders. On 
the other hand, in my role as a small trade association representative, I 
wondered about the potential affect of such vendor selection strategies on 
future developments in computer-based training—namely, whether sound 
development procedures would be compromised in order to fulfill contracts. 

On a parallel front, there exists a general consensus that Computer- 
Based Training (CRT) will be big business in the future (e.g., Training 
magazine's Computer Literacy Week, September 1983 in New York), and the 
field is attracting many entrepreneurs (e.g., August 1 983 issue of 
Training: "How to Make it Big in the Computer-Literacv Training Biz"). 
However, this proliferation is not without Its problems in a business 
populated by individual consultants, small firms and a few malor 
Publishers. For example, the publisher of the authoring svstem chosen for 
AT&T's vendor demonstration, WICAT Systems, went pnbtir with a rommon 



- 1 - 



stock offering on July I, 1 983 • Since its initial prire of $18 per share, 
the price as of thin writing is $3.50 (dut.> primarily to significant 
projected losses). Thus, a "shakeout" may t>> occurring in CBT industry 
before real growth comes to fruition. 

Concurrent with the above developments, one can rarely find a popular 
computing magazine, an issue of Trainin g or an *asue of Performance and 
Instruction, without some expert's opinion on how to evaluate 
instructional computing software. Just as the CRT business proliferates, 
so Ho models on courseware evaluation strategies. One purpose of this 
presentation is to examine a number of published models from three 
perspectives in light of the current CRT Jndustrv situation* Two of these 
perspectives were alluded to in the previous discussion — the 
end-user/pro 1ect proposer and the commercial vendor. A third perspective, 
that of the academic (or non-aligned consultant), Is added to provide 
overall balance. 
Th e Present Context 

1 would be remiss in my primary occupation as an educational evaluator 
and researcher, if I did not relate the current work on computer 
courseware evaluation to the evaluation of educational materials in 
general. After all, computer courseware is a specialized product in the 
overall realm of instructional products. If we think about instructional 
products , how is computer courseware different ? What commonali t ies does 
it have? Does the evaluation of computer courseware require special 
expertise different from the evaluation of Ipps technically sophisticated 
1 n«t ructl onal materials? 

While these questions are considered here, It i« not the primary 
latent of this paper to provide definitive anpwers. ^olas (1983) attempted 



to describe major differences In the format I »>e evahiatt.cn of print 
materials and CAI. She pointed out three differences: hardware, its 
associated components and their reliability; the management of evaluation 
especially In collecting detailed formative data; and the target group's 
famlllarfty and ease of Interaction with a computer. Considering these 
aspects certainly Implies that expertise associated with computer hardware 
interacts with the evaluation process. A more general statement on this 
Issue was provided by Scriven (19R0, o. 11), "The medium modifies the 
message." Tn his discussion of the evaluation and effects of current word 
processing capabilities on instructional research, Scriven (l9R0 f p. 12) 
t a iked about the dif f 1 cul tv of select I ng sof tware : 

" . . . ^nd they must decide whether and which 
ones to purchase for L;ic-ir own or their employ- os 
ube. This is a formidable task. The field is 
functionally a nd te chno loglca L ly m ore comp l e x 
than any consumer goods area. Being an Informed 
purchaser here is perhaps two or three times harder 
than in the automobile field, for example ..." 
( emphasis added) 

Even Robert Stake, whose writings have changed the course of evaluation 
toward the humane and "responsl ve" t Is swaved hy the potential changes to 
hr brought by computers in education when he states that manv educators 
"are outte insensitive to the revolutionarv changes in the nature of 
knowledge [to come]. There is little appreciation for the upcoming 
replacement of print media by electronic media and hew rmirh that will 
change what is considered 'knowledge 1 " , (Stake, 1981). 

Possibly the differences between traditional media and electronic 
media are more profound considering our rather limited history with CUT. 



The implications for evaluation of rmulla of both type* should parallel 
these differences. Except for special emphasis on technical features, 
however, present evaluation strategies have not tended to make such 
distinctions. I shall examine our current status in the rest of this 
paper. 

Method of Invest 1 gat ion 

While this investigation has as its focus on the current approaches to 
computer courseware evaluation, there are probably three levels of 
courseware that mav be considered: program, course, or lesson. It was 
decided that course level was most appropriate, i.e., a big enough prolect 
to assume substantial commitment on the part of potential users. However, 
the majority of literature reviewed for this study did not make this 
distinction. The extent and intensity of review would seem to depend upon 
thv level targeted for analysis. For the most part, I have chosen to stay 
at the course level, but the consulted documents vary across levels. 

A three-stage process was utilized in this investigation. First, a 
list of "primary" sources of computer cour beware evaluat Ion were located 
and analyzed. From this analysis, a list of criteria (or criterion 
elements, e.g. human factors operation) was derived and cross-referenced 
with the primary sources. Thirdly, a checklist of the resulting criterion 
elements was formulated and mailed to select samples of end* users 
(contract-agencies), vendors (courseware developers) and academic based 
educators or facultv. 

The list of criterion elements was grouped Into three main or global 
categories: Use, OuaHty and Efficiency. These global categories were 
chosen mainly as a means for separating the .1 nd 1 vldual elements. As a 
secondary consideration, it was hypothesis d that certain criterion 



elements would receive greater Importance depending upon the perspective 
of the evaluator. For example, Quality might he assumed of greatest 
concern to the Academics. 



RESULTS 

Source Documents and Criteria Identification 

The twelve sources reviewed for this qtudy are listed tn F.xhibit 1. 
Documents with particular emphasis on computer courseware evaluation are 
listed as numbers 3 through 11. Most of these nine documents were 
suggested by colleagues, cited in other literature or selected on the 
basis of current or potential impact. Four of these nine are directly 
related to computer-based training (ChT); others were primarily designed 
for educational or academic audiences, although generalizations and 
transfer to CBT can be assumed on many topics. 

The first two documents (numbers I and 2) represent evaluatl/e 
strategies for educational products in general. Both Scriven's checklist 
and Did 's chapters are highly regarded sources, and they are cited often 
in other literature. The last reference (number 12), on some special 
application software, was chosen to represent the software field in 
general. The quite extensive guidelines used bv Hsu and Mitko (reference 
"12) were based upon criteria originally published in Inf o World. 

The oblective of the review of evaluation stratetr1.es as described or 
imbedded within these sources was to identify a set of criteria 
comprehensive enough to include most of what was discussed yet short 
enough to be manageable. Obviously some sources emphasized some topics 
over others; however, most authors made some claim to state that their 
ideas were comprehensive. Results of this review are given in Exhibit 2 
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where sources are cross-classified with the content analyzed criteria. 
These criteria are purposely phrased in short description terms* With 
some exceptions, most can he applied to educational products generally. 

Another &teo in the process was to classify the specific criteria 
according to three general or global criteria— Use, Duality and 
Efficiency. Tn Exhibit 2 f an f X f indicates that either a discussion or 
some mention of a particular criterion element was given in that source. 
Two exceptions to this are a more detailed list of Instructional 
Strategies (B9) and Demonstrated performance effectiveness (BID where a 
number Indicates how many subtopics were considered (each out of a 
possible eight). 

The data in Exhibit 2 provide only a general indication whether or not 
a given criterion element was discussed. Certain authors were obviously 
interested in some topics more than others, and their guides were not 
necessarily complete for a comprehensive courseware evaluation. For 
example 9 the IRS CBT Starter Kit was developed to provide new or potential 
CBT users with guidelines to be considered. These authors placed more 
emphasis on examining use and cost, and they were less concerned about the 
particular quality of a product. For thoroughness on the topic of Use, the 
CUT Starter Kit and the Kearsley book are recommended to decide whether or 
not CP" is a feasible alternative to present training delivery. 

With regard to the examination of Quality, the sources recommended 
include the Steinberg book, the EPIE Guidelines, and the Reeves and Lent 
paper. A thorough discussion of the category titled Demonstrated 
performance effectiveness is given in the Scriven document. No other 
reference cited was as thorough in this area as his checklist. 

- 6 - 
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The discussions on Efficiency or cost is sketchier in the literature 
than discussions of Mse or Ouality. The Kearsiev book mentions a large 
number of facets with regard to Efficiency, bit better guidelines in terms 
of implementation and use are provided in the IRS publication and the 
Reeves and Lent Paper. Because of present limited information, future 
research might be concentrated in this area. It seems reasonable to 
conclude that Vendors, Users and Academics would all be Interested in 
better studies on the cost and efficiency in using CBT, if its projected 
future is at all consistent with many of the current claims being made. 
Checklist and Survey Results 

The criteria list of Exhibit 2 was translated into a checklist and 
mailed to a small, select, sample of reactors* The individuals contacted 
were asked to provide their opinions based upon. their perspective as a 
computer courseware vendor, end-user (contract agent), or academic 
evaluator /developer. Of the fourteen individuals contacted, twelve 
responded, and the ten who allowed use of their names are listed in 
Kxhibit 3. Two academic respondents refused permission. This sample (five 
Vendors, four Users and three Academics) is not claimed to be 
representative of all such individuals included in each group. Instead, 
the purpose of the survey was to provide expert opinion on the initial 
validity and utility of the criteria list. 

For each criterion element, each respondent was asked whether or not 
it should be required in the evaluation of a given computer courseware 
product. Secondly, they were asked to rate the importance of each element 
on a five-point acale of high to low. The checklist contained the same 
labels aa in Exhibit 2 including the subcategories of Instructional 
strategies (B9) and Demonstrated performance effectiveness (BU). Results 
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for the total group and for Vendors, Users and Academics are displayed in 
Exhibit 4. 

The total sample results (n-12) are provided in the first two column 
sets at the left. As an illustration, the first element under Use should 
be read as "Should a Basic description of the courseware be requir e d . " 
Eleven respondents Indicated it as required , and one indicated as not 
required. Under Importance , a five-point scale of high to low, nine gave 
it a rating of highest importance, one in the next highest category, one 
for the middle category and none in the lowest two (also one person did 
not respond to importance). To the right, these responses are broken down 
by group where the 'R' column shows how many Indicated as required (e.g., 
4/5 means 4 out of 5 who 1 responded). Similarly, the 'V column shows 
Importance ratings where the five response category results are given In 
descending order from most Important. 

A criterion element that appeared near the too of the Importance list 
for Vendors, Users and Academics for the general category of Use was 
Required equipment (6). The next two highly rated elements were 
Documentation (8) and Basic description (1) of the course. Although 
Target groups specifications (2) was indicated as required, Its Importance 
compared to the above three elements was lower. The least Important 
elements, on the other hand, were Match to needs in general (5), 
Dissiminatlon/avallabillty (7), and Longevity (13). In making comparisons 
across the three groups, Academics were more concerned than Vendors and 
Users about the Reliability of operation (10), Human factors operation 
(12) and User organization support (14). Vendors were more interested than 
Users In Reliability of operations (10). In no case wer<» Users more 
interested than Vendors or Academics for any of the Use categories. For 
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Use in general, there was very little differentiation among the groups. 
It would remain to be seen whether or not these results would hold true 
across larger samples. 

The most important category under the heading Ouality was Content 
accuracy (1). Another consistent component across all three groups was 
the Overall instructional effectiveness of the courseware, a suhcategory 
under Demonstrative performance effectiveness (11). Least important 
categories across all three included the Description of rationale (3) and 
surprisingly for Vendors and Users, the Specific match to target needs 
(A). Also of lesser importance to Vendors and Users was the specific 
categories of performance' effectiveness (11). Looking at the differences 
across groups ,^ Users were more interested than Vendors or Academics in a 
Manual for operation of courseware (8) and Lesson and course management 
(12). Vendors, on the other hand, were more Interested than Users in 
Content (skills and adequacy) (2), Learning objectives (5) and Technical 
features (10). Academics were more Interested than Vendors and Users in a 
specific Match to target needs (A), Instructional strategies (9) and 
specific elements of performance effectiveness (11). 

The general ratings for the third part of the questionnaire on 
Efficiency indicated lower importance than many Use or Oualttv areas. 
This mav have been somewhat surprising. For required components, the last 
four categories received fairly high proportions; importance, however, was 
generally low. The most important category across the three groups was 
Cost of acquisition or development (1). Criteria of lesser importance were 
Cost effectiveness resultsO) and Cost benefit detailed analysis (A) 
across all three groups. It appeared, in general, that Vendors were more 
concerned with potential cost factors than were either Users or Academics. 
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Vendors were more Interested In Cost effectiveness results, Cost benefit 
analysis and Management resources needed than were Users. In none of the 
eight categories did Users rate an element more Important then did 
Vendors. 

DISCUSSION 

Earlier In this paper, the question was raided as to whether or not 
computer courseware evaluation was sufficiently different from the 
evaluation of other media-based training. Information relating to that 
question may be derived by looking at the top five criteria across 
categories for the total sample: 

Adjusted Score 



1. Content Accuracy (Rl) 100 

2. Required Kqrlpreent (A6) 95 

3. Documentation (A8) 92 

4. Basic Description (Al) 87 

5. Cost of Acquisition/Development (CI) 82 



The adjusted score is an Importance rating weighted by the number of • 
responses checked as "required".* A score of 100 means that all 
respondents checked the element as "required" and all assigned it the 
highest importance rating. Only two of these five are media 
dependent — Required equipment and Documentation — and one may reasonably 
argue that they are more critical with computers than with other media. 
For comparison , here are the next six: 

Adjusted Score 



6. Seivlce (training to operate) (C6) 76 

7. Content (prerequisite skills, topic 

adequacy) (B2) 75 

8. Overall instructional effectiveness 

(Rllb) 73 

9. Maintainability (Extended support) 

(C7) 73 

10. Equipment compatibility (A9) * 70 

11. Learning oblectives described (R5) 70 
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Three of these— Service, Maintainability and Equipment compatibility— are 
much more specific to computers than to other media. It appears that the 
"medium modifies the evaluation**, to paraphrase an earlier quote. Thus, 
knowledge of computer hardware and software (including documentation) are 
Important aspects for doing an adequate evaluation of courseware. This 
should he no surprise to any readers. 

A second point to be gleaned from the above data is that no global 
category dominates, I.e., Use, Ouallty and Efficiency are all well 
represented. Additionally, comments from respondents indicated only minor 
revision of the checklist. Most revisions were associated with clarity of 
some terms, very little on the comprehensiveness or completeness of what 
wis presented. 1 

Finally, It was expected that Vendors, Users and Academics would 
differ considerably on one or two global categories (Use, Quality or 
Efficiency). Major differences were not noted, and this may in part be. due 
to the small size of the present sample. In general, It was found that 
Users were easier to please, i.e., less rigorous than Vendors or 
Academics. Academics were predictably more concerned with specific 
Instructional strategies and effectiveness than were others. Generally, 
Vendors were more like the Academics in their preferences for criteria. 

Perhaps this study could be expanded to wider representation to 
confirm or discount a num^Tr of present conclusions. Such an expansion 
would serve at least two purposes. *lrst, the results would orovide good 



Importance Rating Mean 



9 Checked Required 



Adj. Scr. ■ 



X 



X 100 



5.0 (Maximum rating) 



12 (Total sample) 
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market research data on competitors (if you 1 re a vendor) f what other Users 
are looking at and how to please Users with products* Secondly, the 
training of future computer courseware developers may be shaped according 
to Industry demands and standards. At least, attention should be paid to 
those elements viewed as important considerations, including research on 
those criteria less understood or used. 
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EXHIBIT 2 (Part A - Use) 
CRITERION ELEMENTS vs, DOCUMENTS MATRIX 



SeaivEN Die. 

Checklist, General 

6eneral Formative 

Educational And 

Product Suhmative 



Sax 
NSPI 

Kearsley Steinberg Paper a 
Book Book 



NCTH 

Article Guidelines 



HlCRO IRS REEVE9 

SIFT Blum CBT And Yeaqer 

Evaluators EPIE Starter Lent CBT 

Guide Paper Kit Paper Workshop 



Hsu 
And 
Nitko 
Special 
Application 
Software 



USE (Is It Usepul Por Hv Needs?) 




1. 


Basic Description 


X 


2. 


Target (jpoup 


X 


3. 


General Context 


X 


t». 


General Purpose/Application 




5, 


Match to Needs 


X 


6. 


Required Equipment 




7. 


Narket/Dissemination (Avai lability) 


~ { 


8. 


DOCIf^NTATION 




Q, 


Equipment Cohpatibility 




10. 


Rf LIABILITY OP ^PERATlON 


" X 


11, 


SCHEDULING/ DEl fVERV 




12. 


Hi'wan Factors Operation _ _ 




15. 


lONfiEVlTV r SMEL^ LlPE) 




!1. 


•l5S» CpfiAN I NATIONAL SuPPTRT 





X 

X 
X 

x" 

X 

J 

X 
X 



X 
X 

A 
V 

X 
X 

X 



X 
X 
X 
X 
X 
X 



X 
X 

x_ 

X 
X 
X 
X 
X 
X 
X 
X 
X 
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EXHIBIT 2 (Part B - Quality) 
CRITERION ELEMENTS vs. DOCUMENTS MATRIX 



CftlTEPlA 

QUALITY (Is I i Any Good?) 

1. Content Accuracy 

2. CoNTfNT (Prc-skills, Topic Adequacy) 

3. Rationale 

1. Specific Match to Target Needs 

5. Learn inc; Objective* 

6. Instructional Technique 

7. Consist*".:*: Technique to Q&jECTwts 

8. Manual 

*9. Instructional Strategies W 

'0. TrcMNicAi Ffaturfs (Graphi cSj Touch ) 

il. Demonstrated Performance Effectivfness (8> 

■ 2. Lesson and Cqursf Mana/senent 



Scri yen 
Checklist, 
General 
Educational 
Product 

X 

X 

x 



Dick 

fiENERAL 

Formative 

And 
Summative 

X 

X 

X_ 

X 
X 

X _ 

X 
X 

6 _ 
X 

X 



Kearslet 
Book 



Steinberg 
Book 

X 
X 

_x_ 

X 
X 

X ^ 

X 

7 

X 

X 



Sax 
NSPI 
Paper * 
Article 



NCTM 
Guidelines 

X 
X 

_ X 

X 
X 



Micro 
SIFT 
[valuators 
Guide 

X 

• X 



X 

J. 

X 
X 

X 

2 



Blum 
EPIF. 
Paper 



X 
X 

X 
X 
X 
X 

±. 

X 

I 

X 



Sbs 

CBT 
Starter 
Kit 



2_ 
X 
6 
X 



Reeves 
And 
Lent 

Paper 



X 
7 
X 



Yeager 
CBT 

NORKSHOP 

X 
X 



X 

X 

x" 

-J. 

X 



Hsu 
And 

NlTKO 

Special 
Application 
Software 

X 
X 



X 

.5. 
X 

? 

y 



$FF ^DDENDUM 
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EXHIBIT 2 (Part C - Efficiency) 
CRITERION ELEMENTS vs. DOCUMENTS MATRIX 



Criteria 

EFFICIENCY (What Doe* It Cost?) 

1. Cost or Acquis it i on/Of vf.lopmfnt 

2, COMPONENT RtSO'JRCES 

5, Cost Cpffctivcnss Results 

CoST-BtNFMT DfTA tLED AnAIYSIS 

5. Cost Implementation 

6. SE«via 'Training) 

7. ]n t a i nab i l i rv - F/TFNDrc Support 

8. flANAiiMfrtr RESOURCES NEEDE? 



SCRIVEN 

Checklist, 
General 
Educa> zonal 
Product 



Dick 

General 
Formative 

And 
suhmative 

X 

X 



Sax 
NSPI 

Kearsley Steinberg Paper ft NCTrl 
Book Book Art icte Guioel ines 



Micro 

SIFT Blum 

Evaluators EP1E 

Guide Paper 



ffiS 
CBT 
Starter 
Kit . 

X 
X 

_x_ 

X 
X 
X 
X 



Reeves 
And 
Lent 
Paper 

X 
X 
X 

" X 
X 



Yeagcr 
CBT 
Workshop 

X 
X 



Hsu 
And 
Nitko 
Special 
Application 
Software 
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EXHIBIT 2 - ADDENDUM 



Criterion Element 

B 9 Instructional Strategies 

a, presentation 

b. locus of control ♦ 

c, interface with related material 

d. questions 

e. responses 

f, feedback 
6. branching 
h, remediation 

Criterion Element 

B 11 Demonstrated Performance Effectiveness" 

A, TRUE FIELD TRIALS WITH TARGET GROUP 

B, CRITICAL COMPARISONS WITH ALTERNATIVES 

C, LONG TERfJ RESULTS 

D, SIDE EFFECTS 

E, ASSOCIATION OF CAUSE TO TREATMENT 

F, RISKS/ MORAL QUESTIONS 

G, STATISTICAL SIGNIFICANCE 

Hi OVERALL INSTRUCTIONAL EFFECTIVENESS 
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EXHIBIT 3 
RESPONDENTS TO CHECKLIST 



Ve ndors 

Bev Coskl'noglu Duosoft Corporation 

Savoy, Illinois 

Andy Gilbert Duosoft Corporation 

Savoy, Illinois 

Richard Lent Digital Equipment Corporation 

Turlington, Massachusetts 

Tony Phelan Global Information Systems Technology 

Champaign, Illinois 

Tom Schaefges Courseware Applications 

Savoy, Illinois 

US£R£ 

Robert Bodine Safeguard Business Systems 

Blue Bell, Pennsylvania 

Chuck Geigner -State Farm Insurance Companies 

Bloomington, Illinois 

-E5 May Digital Equipment Corporation 

Bedford, Massachusetts 

ireg Sharp Southeast Bank Corporation 

Miami, Florida 

Academic (to acknowi fdrf q^i y) 

I. Allan Avner Computer-Based Education Research Laboratory (cerl) 

University of Illinois ,• 
Urbana, Illinois y y 



9 

ERIC 
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Part A, Use 

Criterion Element 

1. Basic Description 

2. Target Group Specification 
5. General Context For Use 

4, General Purpose/Application 

5. Match To Nefos In General 

fi. Required Fouipment 
# 

7, Napket/ ? )es semination 
(Availability) 

H. Documentation 

9, Equipment Compatibility 

10. Reliability of Operamon 

11. SCMEOULINO/OfLlVERY 

1?. Human Factors Operation 
H. Longevity (shelf life) 
14. User Arganuation Support 



TOTAL 



INHIBIT 4 
RESPONDENT RESULTS 

(Is this course/lesson useful for my needs?) 



Not Importance 
Required Required (High-Low) 



ERIC 



n 

7 

T 



12 

" r 
ii 

10 

IT 



7 

T 

8 



1 
1 

V 



2 
*1 
7 
5 

3 



9 J 1 - - 
15 2 1J 
3 - 6 - - 



- 1 3 1 3 
2 2 3 2 2 



Vendors 

(5) 



3 2 3-1 3/; 

11-22 1/5 

"l"- 2 2 3 l /? 

9 1-1- 5 , 5 
5 2 3 - - 

~7 7~-T i 

2 3-2- 



2 7, 



3 1 3 J 2 . 7 



-/c 
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1 . 



s 1 . 

2 1.1 
3 



1 



1 . 



Users 
(4) 



2 /a 



ll 
2 . 



1 . 



1 . 



2 . 



1 . 



1 . 



2 . 



Academics 
(3) 

R I 



2/, 



? 7, 



2 . 
2 . 
2 . 



1 . 



V, 



2 /, 



1 
1 
3 

Y 

3 
1 
3 

1 . 

2 . 



1 



1 . 



26 



EXHIBIT H (Com'.) 



(IS THIS COURSE/LESSON AMY 6000?) 



Part B. Quality 
1 Mm low Element 
1. Content Accuracy 



TOTAL 

Mot 

Required Reouired 

12 



Importance 
(High-Low) 



12 - - 



Vendors 
(5) 



5 „ 



Users 
tt) 



Academics 

(3) 



I 



3 . 



2. Content (prerequisite skills, 

topic adequacy) 

3. Rationale Description 



<t. Specific Hatch to Target Needs 



5. Learning Objectives Described 



6. Instructional Technique 



7. Consistency t Technique to 

Objectives 

8. Manual for Operation 



9. Instructional Strategies 

(8 components) 

10. Technical Features (graphics, 

sound, touch, etc.) 

11. a. Demonstrated Performance 

Effectiveness 

b. Overall Instructional 
Effectiveness 

12. Lesson and Course Management 



11 
5 



H 5 1 1 - 
1113 2 



>/ 5 



10 



10 



2 3 2 1 1 



5 3 11 



1 5 4 - 1 



5 ' 5 1 



\ 2 



10 



3 3 4 1- 



2 2 6 



71 23 /43/21/14/10/ - 



3 2 5 



IH 50 / 9/23/19/10/ 2 



30/ 



15 



10 



10 



6 3 2 - 



3 2 5 1 - 



% 2 



\ '2- 



1/. 1 



1 . 



3 



1 



\ »1. 



2 . 

2 . 

122 

1 . 

2 . 



78 



1 . 



f /fl 



17/ 



'21 



1 i 

^ a. 



2 /, 1 1 



^3 ■ - 1 . 

22 /„ °18 r . 



1 . 



1 



1 . 



1 . 



1 . 



ERIC 



27 



28 



Part C. Efficiency 



Criterion Element Not 

Required Required 

1. Cost of acquisition/development 9 2 

2. Component Resources Needed 8 3 

3. Cost Effectiveness Results 5 7 
Cost/Benefit Jetailed Analysis 5 6 

5. Cost of Implementation 10 2 

6. Servitf: (Training to Operate) M 

7. Maintainability-Extended Support U 1 
3. Management Resources Needed 10 2 
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EXHIBIT 1 (Cont'.) 

(What does lesson/course really cost?) 

Vendors u *f" s Academics 

(5) (,,) (3) 

Importance r l R I R j 

(High-Low) 

7 2 1 " " \ % 1, 2 1.. 

621 -- % 3 ii._ % v 2 i.. 

512 J ' ^ 'll.. % \ 

3 5 2 " ' % "'2._ % 2 L. 2 / 3 



2 «l 1 1 1 



1 



1 



5 /c " 1 2/1 3,2 

- 1 



1 
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